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ABSTRACT 

The  National  Agricultural  Statistics  Service  (NASS)  has  expressed  concern  about  the  cost 
effectiveness  of  survey  editing.  This  is  particularly  true  since  1997  when  NASS  assumed  the 
responsibility  for  the  census  of  agriculture  for  which  hand-editing  of  all  questionnaires  is  not 
feasible.  In  the  1997  Census  of  Agriculture,  all  reported  data  were  keyed  and  then  edited  by 
computer.  There  was  no  attempt  to  sort  the  records  according  to  importance  of  review.  Records 
flagged  for  review  during  the  computer  edit  were  reviewed  in  the  order  in  which  they  were 
keyed.  Thus  time  was  spent  reviewing  records  which  had  little  impact  on  any  county  estimates. 
Some  important  records  which  should  have  been  reviewed  more  thoroughly  probably  were  not, 
due  to  time  constraints. 

Survey  estimates  can  be  improved  by  priority  sorting  the  records  so  that  those  records  that  are 
more  likely  to  contain  errors  which  will  have  a  significant  effect  on  the  county  estimates  are 
reviewed  first.  This  priority  sorting  of  records  for  review  is  accomplished  by  first  applying  a 
score  function  and  then  sorting  the  records. 

This  paper  presents  an  evaluation  of  three  score  functions  thought  to  be  most  suitable  for  data 
collected  by  NASS.  The  ultimate  goal  is  to  reduce  the  labor-intensive  manual  review  of  data 
without  damaging  data  quality.  The  results  indicate  that  using  a  score  function  presents  an 
improvement  over  the  previous  way  of  reviewing  census  records.  The  paper  provides  detailed 
information  on  each  score  function,  the  methodology  followed  to  evaluate  each  function,  the 
recommended  score  function  to  use,  and  some  thoughts  for  further  research. 
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SUMMARY 


The  National  Agricultural  Statistics  Service  (NASS)  has  expressed  concern  about  the  cost 
effectiveness  of  survey  editing.  This  is  particularly  true  since  1997  when  NASS  assumed  the 
responsibility  for  the  census  of  agriculture  for  which  hand-editing  of  all  questionnaires  is  not 
feasible.  For  these  reasons,  NASS  is  conducting  an  evaluation  of  its  current  editing  process.  The 
ultimate  goal  is  to  reduce  the  labor-intensive  manual  review  of  data  without  damaging  data 
quality. 

Survey  estimates  can  be  improved  by  priority  sorting  the  records  so  that  those  records  that  are 
more  likely  to  contain  errors  which  will  have  a  significant  effect  on  the  county  estimates  are 
reviewed  first.  This  priority  sorting  of  records  for  review  is  accomplished  by  first  applying  a 
score  function  and  then  sorting  the  records.  A  score  function  produces  a  score  for  each  record 
such  that  (theoretically)  the  higher  the  score  the  more  important  it  is  to  review  that  record.  The 
score  may  be  a  function  of  the  magnitude  of  the  record  (e.g.  for  each  variable,  the  percentage  that 
record  contributes  to  the  sum  over  all  records).  It  may  also  depend  on  the  amount  that  the  record 
has  changed  since  the  previous  reporting  period. 

Three  candidate  score  functions  were  chosen  for  evaluation  in  ordering  the  farm  records  in  the 
next  census.  The  difference  score  function  uses  the  absolute  difference  between  variables,  the 
ratio  score  function  uses  the  ratio  of  variables,  and  the  magnitude  score  function  is  the  same  as 
the  difference  function  except  it  does  not  use  record-level  data  from  the  previous  period.  The 
1997  Census  of  Agriculture  data  from  Michigan  and  Arkansas  were  used  to  evaluate  the  score 
functions.  Michigan  data  were  selected  for  the  evaluation  largely  due  to  the  diversity  of 
agriculture  in  the  state.  Arkansas  data  were  chosen  since  it  has  a  different  set  of  farm 
characteristics  than  Michigan.  Characteristics  such  as  farm  acreage,  total  value  of  products  sold, 
acres  harvested,  and  inventory  among  others  were  used  to  score  the  records  in  both  states. 

The  results  showed  all  three  score  functions  being  an  improvement  over  the  actual  review  order. 
The  difference  and  magnitude  score  functions  are  comparable  when  determining  review  order  for 
records  with  historical  data.  Also,  both  score  functions  outperformed  the  ratio  function.  When 
records  were  processed  in  batches  the  score  functions  showed  some  loss  in  efficiency,  however 
all  three  functions  still  presented  an  improvement  over  the  actual  review  order.  As  a  result  of  the 
analysis,  the  recommendation  is  to  use  the  magnitude  score  function  for  the  2002  Census  of 
Agriculture. 


I  Introduction 

Historically,  the  focal  point  of  the  editing 
process  was  on  correcting  all  or  as  many 
errors  in  a  survey  as  possible.  However,  in 
recent  years  the  survey  editing  process  has 
been  the  focus  of  serious  evaluations  and  re¬ 
engineering  in  many  organizations.  This 
change  in  focus  was  primarily  because  of  the 
high  cost  associated  with  editing  all  records 
instead  of  concentrating  on  those  records 
that  could  significantly  improve  survey 
results  (Latouche  and  Berthelot,  1992; 
Lawrence  and  McDavitt,  1994). 

To  address  this  issue,  survey  researchers 
have  come  up  with  techniques  such  as 
significance  editing  and  selective  editing. 
These  methods  consist  of  reducing  and 
prioritizing  the  editing  workload  by 
examining  individual  records  based  on  their 
potential  impact  on  the  estimates.  Basically, 
the  new  view  on  editing  is  to  prioritize  the 
order  in  which  the  records  are  edited  so  that 
the  most  important  records  are  addressed 
most  rigorously  (Lawrence  and  McDavitt, 
1994).  By  ignoring  records  of  less  aggregate 
impact,  editing  resources  can  be  reduced 
significantly.  Respondent  burden  is  also 
reduced  since  the  amount  of  respondent 
recontact  is  minimized.  This  will  in  tum 
lead  to  an  improvement  in  the  overall  quality 
of  the  final  estimates  since  resources  are 
used  for  a  more  extensive  follow-up  of  the 
most  important  records  (Lawrence  and 
McKenzie,  2000).  To  prioritize  the  order  in 
which  records  are  reviewed  requires  the  use 
of  a  score  function  (Latouche  and  Berthelot, 
1992). 

The  National  Agricultural  Statistics  Service 
(NASS)  has  expressed  concern  about  the 
cost  effectiveness  of  survey  editing.  This  is 
particularly  true  since  1997  when  NASS 
assumed  the  responsibility  for  the  census  of 
agriculture  for  which  hand-editing  of  all 


questionnaires  is  not  feasible.  For  these 
reasons,  NASS  is  conducting  an  evaluation 
of  its  current  editing  process.  The  ultimate 
goal  is  to  reduce  the  labor-intensive  manual 
review  of  data  without  damaging  data 
quality. 

In  the  1997  Census  of  Agriculture,  all 
reported  data  were  keyed  and  then  edited  by 
computer.  The  format  and  simple  edit 
procedure  flagged  selected  problem  cases  for 
review.  However,  there  was  no  attempt  to 
sort  the  records  according  to  importance  of 
review.  Records  flagged  for  review  during 
the  computer  edit  were  reviewed  in  the  order 
in  which  they  were  keyed.  Thus  time  was 
spent  reviewing  records  which  had  little 
impact  on  any  county  estimates.  Some 
important  records  which  should  have  been 
reviewed  more  thoroughly  probably  were 
not,  due  to  time  constraints. 

Survey  estimates  can  be  improved  by 
priority  sorting  the  records  so  that  those 
records  that  are  more  likely  to  contain  errors 
which  will  have  a  significant  effect  on  the 
county  estimates  are  reviewed  first.  This 
priority  sorting  of  records  for  review  is 
accomplished  by  first  applying  a  score 
function  and  then  sorting  the  records. 

This  paper  evaluates  three  widely  used  score 
functions  thought  to  be  most  suitable  for 
data  collected  by  NASS.  The  analysis  is 
done  using  1997  Census  of  Agriculture  data 
from  Michigan  and  Arkansas.  Section  II 
provides  detailed  information  on  each  score 
function.  Section  III  describes  the  procedure 
followed  to  evaluate  each  function.  Sections 
IV  and  V  present  the  results  for  Michigan 
and  Arkansas,  respectively.  Section  VI 
offers  the  recommended  function  to  use,  and 
Section  VII  provides  some  thoughts  for 
further  evaluation. 
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Score  Functions 

A  score  function  produces  a  score  for  each 
record  such  that  (theoretically)  the  higher  the 
score  the  more  important  it  is  to  review  that 
record.  The  score  may  be  a  function  of  the 
magnitude  of  the  record  (e.g.  for  each 
variable,  the  percentage  that  record 
contnbutes  to  the  sum  over  all  records).  It 
may  also  depend  on  the  amount  that  the 
record  has  changed  since  the  previous 
reporting  period. 


the  corresponding  term  of  the  series  is  set 
equal  to  zero. 


The  second  candidate  score  function, 
attributed  to  Hidiroglou  and  Berthelot 
(1986),  uses  the  ratio  of  variables.  The 
scores  it  produces  will  be  referred  to  as  ratio 
scores.  The  calculation  of  a  record’s  ratio 
score  is  as  follows: 


r 

i 


where  x,  i,  and  t  are  as 
described  earlier 


Three  candidate  score  functions  were  chosen 
for  evaluation  in  ordering  the  farm  records 
in  the  next  census.  Although  the  three  score 
functions  evaluated  use  multiple  input 
variables  to  produce  a  score,  none  take  into 
consideration  the  covariance  of  the  input 
variables.  For  example,  in  many  places  one 
would  expect  cattle  farmers  to  grow  hay  for 
winter  feed.  One  would  further  expect  hay 
acreage  to  increase  as  the  number  of  cattle 
increased.  A  deviation  from  the  typical 
relationship  between  the  two  variables  might 
indicate  that  a  farm  record  contains  bad  data. 
None  of  the  three  candidate  score  functions 
would  detect  such  a  deviation. 


Find  rmi  the  median  of  the  r, ,  and  calculate 

S,  =  1  -  0,„  lri)  ifr,<r,„- 

=  (r,  Ir,,,)  - 1  ‘f  n  >r„u 

£i  =  *s, 

where  =  maxl.r  , , 


Find  Eu , ,  E mi  ,  and  E 3j  —  the  first  quartile, 
median,  and  third  quartile  of  the  E (  —  and 
calculate 


D,  =  \(E, 


The  first  candidate  score  function  uses  the 
absolute  difference  between  variables,  so  the 
scores  it  produces  will  be  referred  to  as 
difference  scores.  It  is  attributed  to  Latouche 
and  Berthelot  (1992),  and  is  currently  used 
in  the  Department  Of  Energy’s  GEAQS  edit 
system.  A  record’s  difference  score  is 
calculated  as 


Here  i  indexes  n  farm  characteristics,  x  is  a 
farm  characteristic,  X  is  the  total  for  the 
characteristic,  and  t  indexes  time  (current  or 
previous  reporting  period.)  When  X(f_]=0, 


The  ratio  score  for  a  record  is 


The  ratio  r.  has  three  undesirable  properties. 
First,  r.  is  undefined  when  xfl  equals  zero. 
But,  if  xit  is  nonzero  then  this  situation  may 
correspond  to  a  significant  increase  in 
characteristic  xr  This  problem  is  fixed  by 
setting  xit  ]  equal  to  one  whenever  xit  was 
nonzero.  Second,  r.  is  equal  to  zero  when 
xjt  equals  zero  and  xM  was  nonzero.  Thus, 
it  does  not  take  into  account  the  magnitude 
of  x  f_j .  This  problem  is  partially  fixed  by 
setting  x f  equal  to  1  before  calculating  r.. 

If  both  xf  and  xf  l  equal  zero,  the  ratio 
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was  not  calculated  and  did  not  contribute  to 
the  ratio  score.  Third,  r  is  greater  than  one 
when  xit  is  greater  than  x  M  ;  but  r.  is 
between  zero  and  one  when  xjt  is  less  than 
So,  the  distance  from  rmi  for  an  r  to 
be  perceived  as  an  outlier  depends  upon 
which  side  of  rmi  the  r  lies.  This  problem 
is  fixed  by  utilizing  St  and  E t  to  produce  the 
ratio  score. 

The  third  candidate  score  function  is  the 
same  as  the  difference  function  except  it 
does  not  use  record-level  data  from  the 
previous  period  (i.e.,  xf_, ).  The  scores  it 
produces  will  be  referred  to  as  magnitude 
scores.  Similar  to  Luzi  and  Pallara  (1999) 
whose  scoring  technique  extended  the 
selective  editing  approach  to  fit  cross- 
sectional  surveys,  the  magnitude  score 
function  does  the  same  by  using  previous 
data  only  at  the  aggregate-level  in  order  to 
locate  suspicious  observations.  A  record’s 
magnitude  score  is  calculated  as 

i=i 

Here  i  indexes  n  farm  characteristics,  x  is  a 
farm  characteristic,  X  is  the  total  for  the 
characteristic,  and  t  indexes  time  (current  or 
previous  reporting  period).  When  X  f_,=0, 
the  corresponding  term  of  the  series  is  set 
equal  to  zero.  Note  that  totals  are  the  only 
data  required  from  the  previous  reporting 
period.  When  a  farm  record  has  no  record- 
level  data  for  the  previous  reporting  period, 
this  is  the  only  score  function  of  the  three 
where  scores  can  be  computed. 

To  evaluate  these  score  functions  for  the 
census  of  agriculture,  county  totals 
(weighted)  and  unweighted  record-level  data 
were  used  in  calculating  the  three  different 
types  of  scores.  Unweighted  record-level 
data  were  used  because  weights  are  not 


assigned  until  late  in  the  review  process. 
Using  unweighted  rather  than  weighted 
record-level  data  in  the  score  functions  will 
result  in  different  scores  and  a  somewhat 
different  review  order.  However,  the  overall 
effect  should  be  negligible  since  the  census 
weights  are  small. 

in  Score  Function  Evaluation 

The  1997  Census  of  Agriculture  data  were 
used  to  evaluate  the  score  functions.  The 
following  diagram  shows  the  general  flow  of 
census  data  processing  in  1997.  The 
differences  in  the  files  were  mostly  due  to 
the  processes  listed  on  the  right. 

1997  KEYED  FILE 

Computer  edit  process 
y  Failed  edit  review  process 

1997  EDITED  FILE 

Analytical  review  process 

▼ 

1997  FINAL  FILE 

The  1997  keyed  file  contained  census  data 
after  keying  verification.  The  1997  edited 
file  represented  the  data  after  the  computer 
edit  and  failed  edit  review  processes. 

During  the  computer  edit,  records  which 
“failed”  the  edit  were  flagged  for  further 
review  (i.e.,  manual  review  by  an  analyst). 

As  stated  earlier,  flagged  records  were 
reviewed  in  the  order  in  which  they  were 
keyed.  The  failed  edit  review  process 
focused  on  reviewing  data  at  the  micro-level 
(i.e.,  record-level).  Note  that  there  is  no  way 
to  differentiate  between  manual  review 
changes  and  computer  changes  on  the  edited 
file.  The  1997  final  file  represented  the  data 
after  the  analytical  review  process.  During 
this  process,  record-level  data  were  changed 
after  reviewing  state  and  county  totals.  The 
analytical  review  process  focused  on 
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reviewing  data  at  the  macro-level  (i.e., 
totals). 

To  evaluate  the  impact  of  incorporating  a 
score  function  during  the  failed  edit  review 
process,  only  records  reviewed  during  the 
failed  edit  review  process  were  included  in 
the  analysis.  To  obtain  the  necessary  data, 
the  1997  final  and  edited  files  were  matched 
to  the  1992  final  file.  The  resulting  file  was 
subsetted  to  include  only  records  on  the 
1997  final  file  which  were  reviewed  during 
the  failed  edit  review  process.  Records  were 
scored  and  then  sorted  using  each  of  the 
three  candidate  functions.  All  three  score 
functions  used  data  from  the  1997  edited  file 
for  the  x  .  The  ratio  and  difference  score 

U 

functions  also  used  data  from  the  1992  final 
file  for  the  xil_1 .  The  1992  county  totals 
were  obtained  and  used  as  the  X  .  in  the 

i.t- i 

difference  and  magnitude  score  functions. 

As  stated  earlier,  manual  review  changes 
could  not  be  differentiated  from  computer 
changes  on  the  edited  file.  Thus,  the 
effectiveness  of  the  score  functions  was 
evaluated  through  the  use  of  a  review 
adjustment  score  which  measured  changes 
made  during  the  analytical  review  process. 
Data  from  the  1997  final  and  edited  files 
were  used  to  calculate  a  review  adjustment 
score  for  each  record  in  the  subset  described 
above.  A  record’s  review  adjustment  score 
indicates  the  importance  of  reviewing  that 
record  (the  score  is  higher  for  records  with 
significant  analytical  review  changes).  The 
review  adjustment  score  is  calculated  as 

n 

[Shu  - 

/=! 

Here  i  indexes  the  n  farm  characteristics 
being  examined,  x  and  X  are  as  described 
before,  k  corresponds  to  the  1997  final  file, 
and  k-1  corresponds  to  the  1997  edited  file. 


To  reiterate,  the  review  adjustment  score 
takes  into  account  changes  made  only  during 
the  analytical  review  process  of  records 
flagged  during  the  failed  edit  review 
process. 

The  percentage  of  total  review  adjustment 
contributed  by  the  record  was  also  calculated 
to  indicate  the  relative  importance  of 
reviewing  a  record  with  respect  to  other 
records.  The  percentage  of  total  review 
adjustment  contributed  by  the  record  is 

[Shu  -Vi-il^ul* 

i=l  *100 

m  n 

[SSh„u -Xu-,|/ *ul 

;=[  '=i 

where  j  indexes  the  m  records  on  the  1997 
final  file  which  were  reviewed  during  the 
failed  edit  review  process.  Also,  i  indexes 
the  n  farm  characten sties  being  examined,  x 
and  X  are  as  described  before,  k  corresponds 
to  the  1997  final  file,  and  k-1  corresponds  to 
the  1997  edited  file. 

Graphs  1  through  8  in  the  appendix  compare 
the  results  of  scoring  the  records  using  the 
three  proposed  score  functions  against  the 
actual  review  order.  The  cumulative 
percentage  of  the  total  review  adjustment  is 
plotted  along  the  vertical  axis.  The 
horizontal  axis  represents  the  number  of 
records  reviewed  as  specified  by  the  method. 
For  example,  record  reviewed  #10  could 
(most  likely  would)  be  a  different  record 
under  the  actual,  magnitude,  difference,  and 
ratio  score  functions.  Also,  the  scale  of  the 
horizontal  axis  varies  since  all  records  do 
not  have  record-level  data  for  the  previous 
reporting  period.  Thus,  each  line  in  the 
graphs  depicts  the  percentage  of  total  review 
adjustment  for  the  review  order  specified  by 
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each  one  of  these  methods.  Graphs  1-4 
present  the  results  for  Michigan  and  Graphs 
5-8  for  Arkansas. 

IV  Michigan  Data 

Michigan  data  were  selected  for  the 
evaluation  largely  due  to  the  diversity  of 
agriculture  in  the  state.  In  Michigan  there 
were  6,698  records  on  the  1997  final  file 
which  were  reviewed  during  the  failed  edit 
review  process.  Of  these  records,  there  were 
1,496  records  where  data  were  changed 
between  the  1997  edited  file  and  the  1997 
final  file.  Difference  and  ratio  scores  could 
not  be  calculated  using  all  6,698  records 
since  2,644  records  were  not  on  the  1992 
final  file  (i.e.,  no  historical  data).  Table  1 
shows  the  distribution  of  the  reviewed 
records  and  number  of  records  that  were 
actually  changed  by  whether  or  not  the 
record  had  historical  data. 


Table  1:  Michigan  Records  Reviewed  and 
Changed  by  Type  of  Data 


Records 

Number 

Number 

Reviewed 

Changed 

Historical  data 

4,054 

953 

No  historical  data 

2,644 

543 

Total 

6,698 

1,496 

A  total  of  18  variables  were  used  to  score 
the  records  in  Michigan.  These  were  total 
farm  acreage;  total  value  of  products  sold; 
total  cattle  and  calves;  milk  cows;  steers, 
steer  calves,  bulls  and  bull  calves;  total  hogs 
and  pigs;  and  acres  harvested  and  production 
of  the  following:  field  com  for  grain,  field 
com  for  silage,  soybeans,  potatoes,  alfalfa, 
and  winter  wheat. 

It  was  thought  that  the  variables  age,  race, 
and  gender  might  be  useful  for  identifying 
records  with  historical  data  which  should  be 


reviewed.  An  operator’s  gender  and  race 
isn’t  likely  to  change  from  census  to  census, 
and  the  operator  should  age  about  five  years. 
However,  there  were  only  a  few  records 
where  at  least  one  of  these  variables  was 
changed  during  the  analytical  review 
process.  Furthermore,  it  is  difficult  to 
properly  score  categorical  variables. 
Therefore,  these  variables  were  not  included 
in  the  analysis. 

A.  Michigan  Records  With  Historical 
Data 

There  were  4,054  reviewed  records  which 
had  histoncal  data.  Of  these  records,  953 
were  actually  changed. 

Graph  1  shows  how  the  four  different  review 
orders  identified  records  with  analytical 
review  changes  which  significantly  affected 
county  estimates.  Ratio  scores  were  a 
significant  improvement  from  the  actual 
review  order.  Difference  and  magnitude 
scores  outperformed  ratio  scores  with 
difference  scores  barely  outperforming 
magnitude  scores. 

B.  Michigan  Records  Without 
Histoncal  Data 

There  were  2,644  reviewed  records  without 
historical  data.  These  records  could  not  be 
given  ratio  or  difference  scores  since  they 
did  not  have  data  for  1992.  Thus,  each  of 
these  records  was  assigned  only  a  magnitude 
score.  Of  these  records,  543  were  changed. 

Graph  2  shows  that  the  magnitude  scores 
were  an  improvement  from  the  actual  review 
order. 

C.  Michigan  Records  With  and  Without 
Historical  Data 

As  previously  stated,  of  the  6,698  reviewed 
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records  with  and  without  historical  data, 
1,496  were  changed.  Since  the  difference 
and  ratio  scores  could  not  be  calculated  for 
records  without  historical  data,  the  set  of  all 
reviewed  Michigan  records  had  to  be 
evaluated  using  a  score-combining  scheme. 
Records  with  historical  data  used  these 
methods  while  records  without  historical 
data  were  scored  by  magnitude.  The  scores 
were  “standardized”  as  described  below  so 
that  all  records  were  comparably  scored  in 
the  combined  file.  Difference  scored 
records  were  sorted  in  ascending  score 
order,  and  each  record  was  assigned  a  new 
score  equal  to  the  cumulative  difference 
score  for  that  record  divided  by  the  total  of 
all  difference  scores.  This  procedure  was 
repeated  on  the  records  lacking  historical 
data,  using  magnitude  scores  to  produce  new 
scores.  The  two  sets  of  records  with  new 
compatible  scores  were  then  combined.  The 
same  procedure  described  above  was  also 
performed  for  ratio  scores. 

Graph  3  evaluates  the  three  score  functions 
after  the  score-combining  scheme.  The  thm- 
solid  line  (labeled  difference)  shows  the 
result  of  combining  difference  scores  for 
records  with  historical  data  and  magnitude 
scores  for  records  without  historical  data. 
The  thick-dotted  line  (labeled  ratio)  shows 
the  result  of  combining  ratio  scores  for 
records  with  historical  data  with  magnitude 
scores  for  records  without  historical  data. 

The  thin-dotted  line  (labeled  magnitude) 
shows  the  effectiveness  of  using  magnitude 
scores  for  all  records. 

The  results  show  that  the  ratio  scores  were 
outperformed  by  both  difference  and 
magnitude  scores.  Magnitude  scores  were 
more  effective  than  difference  scores  up 
until  70  percent  of  the  total  review 
adjustment  was  reached,  then  difference 
scores  outperformed  magnitude  scores. 


D.  Michigan  Batch  Processing 

For  the  1997  Census,  records  were 
processed  in  batches.  When  enough  records 
were  keyed  in  a  state,  the  programmers  were 
notified  to  run  these  records  through  the 
computer  edit.  This  was  referred  to  as  an 
edit  batch.  Batches  were  formed 
chronologically  based  on  the  date  of  entry 
into  the  system.  The  number  of  records  in 
an  edit  batch  varied  as  did  the  number  of 
edit  batches  in  a  state.  However,  the  failed 
edit  review  process  did  not  occur  until  the 
entire  state  was  processed. 

For  the  2002  Census,  plans  are  to  perform 
the  failed  edit  review  process  after  each 
batch  is  processed.  In  order  to  evaluate  the 
performance  of  the  score  functions 
considering  the  impact  of  batch  processing, 
the  batch  process  was  simulated  using  1997 
Michigan  data. 

In  Michigan,  records  were  processed  in  two 
batches.  To  simulate  the  batching  process, 
the  6,698  records  were  split  into  two  groups 
with  proportions  equal  to  those  of  the  actual 
edit  batches.  For  the  first  group,  4,627 
records  were  reviewed  and  1,129  changed. 
Similarly,  for  the  second  group,  2,071 
records  were  reviewed  and  367  changed. 
Table  2  presents  the  distribution  of  the 
reviewed  records  and  the  number  of  records 
that  were  actually  changed  by  whether  or  not 
the  record  within  each  edit  group  had 
historical  data. 
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Table  2:  Michigan  Records  Reviewed  and 
Changed  by  Edit  Group  and  Type  of  Data 


Records 

Number 

Reviewed 

Number 

Changed 

1st  edit  group 

4.627 

. 

1.129 

-  historical  data 

2,675 

703 

-  no  historical  data 

1,952 

426 

2nd  edit  group 

2,071 

367 

-  historical  data 

1,379 

250 

-  no  historical  data 

692 

117 

Total 

6,698 

1,496 

Scores  were  calculated  using  the  score¬ 
combining  strategy  described  in  the  previous 
section.  The  records  were  scored  and  sorted 
within  each  of  the  edit  groups.  Graph  4 
presents  these  results. 

For  both  edit  groups,  all  three  score 
functions  presented  an  improvement  over 
the  actual  review  order.  Although  the  second 
group  had  fewer  records,  the  records 
accounted  for  68%  of  the  total  review 
adjustment. 

Scoring  and  reviewing  distinct  groups  of 
records  decreased  the  efficiency  of  the  score 
functions.  Using  the  magnitude  score 
function  as  a  point  of  reference,  scoring  all 
records  together  required  reviewing  2,942 
records  to  find  90%  of  the  total  review 
adjustment  (See  graph  3).  From  Graph  4, 
scoring  the  two  distinct  groups  of  records 
required  reviewing  6,033  records  to  find 
90%  of  the  total  review  adjustment. 

E.  Michigan  Conclusion 

The  graphs  show  all  three  score  functions 
being  an  improvement  over  the  actual 
review  order.  The  difference  and  magnitude 
score  functions  are  comparable  when 


determining  review  order  for  records  with 
historical  data.  Also,  both  score  functions 
outperform  the  ratio  function.  The 
combining  process  introduces  potential 
problems  because  of  uncertainty  about 
which  scores  are  equivalent.  The  method  of 
equating  scores  used  in  this  report,  resulted 
in  the  magnitude  scores  performing  about 
the  same  as  combining  the  difference  and 
magnitude  scores. 

V.  Arkansas  Data 

Arkansas  data  were  chosen  to  evaluate  the 
score  functions  since  it  has  a  different 
variety  of  farm  characteristics  than 
Michigan.  In  Arkansas,  there  were  7,187 
records  on  the  1997  final  file  which  were 
reviewed  during  the  failed  edit  review 
process.  Of  these  records,  there  were  1,308 
records  where  data  were  changed  between 
the  1997  edited  file  and  1997  final  file.  As 
was  the  case  with  Michigan,  the  difference 
and  ratio  scores  could  not  be  calculated 
using  all  7,187  records  since  2,328  records 
did  not  have  historical  data.  Table  3  shows 
the  distribution  of  the  reviewed  records  and 
the  number  of  records  that  were  actually 
changed  by  whether  or  not  the  record  had 
historical  data. 

Table  3:  Arkansas  Records  Reviewed  and 
Changed  by  Type  of  Data 


Records 

Number 

Number 

Reviewed 

Changed 

Historical  data 

4.859 

864 

No  historical  data 

2.328 

444 

Total 

7.187 

1,308 

A  total  of  26  variables  were  used  to  score 
the  records  in  Arkansas.  These  were  total 
farm  acreage;  total  value  of  products  sold; 
total  cattle  and  calves;  milk  cows;  steers, 
steer  calves,  bulls  and  bull  calves;  total  hogs 
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and  pigs;  inventory  and  number  sold  of  the 
following:  layers  20  weeks  old  and  older, 
broilers,  and  turkeys  for  slaughter;  acres 
irrigated  and  production  of  rice;  and  acres 
harvested  and  production  of  the  following; 
field  com  for  grain,  field  com  for  silage, 
soybeans,  cotton,  alfalfa,  and  wheat. 

A.  Arkansas  Records  With  Historical 
Data 

There  were  4,859  reviewed  records  which 
had  historical  data.  Of  these  records,  864 
were  changed. 

Graph  5  shows  how  the  four  different  review 
orders  identified  records  with  analytical 
review  changes  which  significantly  affected 
county  estimates.  All  three  methods  (ratio, 
difference,  and  magnitude)  were  an 
improvement  over  the  actual  review  order. 
Difference  scores  slightly  outperformed 
magnitude  scores.  Both  difference  and 
magnitude  scores  were  a  vast  improvement 
over  ratio  scores. 

B.  Arkansas  Records  Without  Historical 
Data 

There  were  2,328  reviewed  records  without 
historical  data.  For  this  reason,  the 
calculation  of  difference  and  ratio  scores 
was  not  possible.  Thus,  each  of  these 
records  was  assigned  only  a  magnitude 
score.  Of  the  records  reviewed,  444  were 
actually  changed. 

Graph  6  shows  that  the  magnitude  scores 
were  a  tremendous  improvement  from  the 
actual  review  order.  With  the  magnitude 
score  function,  only  50  records  needed  to  be 
reviewed  to  find  90%  of  the  total  review 
adjustment,  while  1,112  records  had  to  be 
reviewed  to  achieve  the  same  mark  with  the 
strictly  chronological  (actual)  process. 


C.  Arkansas  Records  With  and  Without 
Historical  Data 

As  previously  stated,  of  the  7,187  reviewed 
records  with  and  without  historical  data, 
1,308  were  changed.  The  same  score¬ 
combining  strategy  used  with  Michigan  data 
was  followed  for  Arkansas  (see  Section  IV 
sub-section  C  for  a  description  of  this 
scheme).  Graph  7  evaluates  the  three  score 
functions  after  the  score-combining  scheme. 

Once  again,  all  three  score  functions 
presented  an  improvement  over  the  actual 
review  order.  Also,  difference  and 
magnitude  scores  outperformed  ratio  scores. 
At  first,  magnitude  scores  performed  better 
than  difference  scores,  but  then  difference 
scores  performed  slightly  better  than 
magnitude  scores.  However,  this  shift  in 
performance  happened  only  after  90%  of  the 
total  review  adjustment  had  been  obtained. 

D.  Arkansas  Batch  Processing 

In  Arkansas,  records  were  processed  in  three 
batches.  To  simulate  this  process,  the  7,187 
records  were  split  into  three  groups  with 
proportions  equal  to  those  of  the  actual  edit 
batches.  For  the  first  group,  1,972  records 
were  reviewed  and  252  changed.  Similarly, 
for  the  second  and  third  groups,  2,646  and 
2,569  records  were  reviewed  and  361  and 
695  changed  respectively.  Table  4  presents 
the  distribution  of  the  reviewed  records  and 
the  number  of  records  that  were  actually 
changed  by  whether  or  not  the  record  within 
each  edit  group  had  historical  data. 
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Table  4:  Arkansas  Records  Reviewed  and 

Changed  by  Edit  Group  and  Type  of 
Data 


Records 

Number 

Reviewed 

Number 

Changed 

1st  edit  group 

1,972 

252 

-  historical  data 

1,224 

168 

-  no  historical  data 

748 

84 

2nd  edit  group 

2,646 

361 

-  historical  data 

1,686 

214 

•  no  historical  data 

960 

147 

3rd  edit  group 

2,569 

695 

-  historical  data 

1,949 

482 

•  no  historical  data 

620 

213 

Total 

7,187 

1,308 

Scores  were  calculated  using  the  score¬ 
combining  strategy  described  in  Section  IV 
sub-section  C.  The  records  were  scored  and 
sorted  within  the  edit  groups.  Graph  8 
presents  these  results 

For  the  first  group  of  records  processed,  all 
score  functions  performed  about  the  same. 
However,  during  the  processing  of  the 
second  group  of  records,  the  difference  and 
magnitude  scores  outperformed  both  the 
ratio  and  actual  scores.  There  was  also  a 
noticeable  loss  in  efficiency  for  the  ratio 
scores.  For  the  last  group  of  records,  the 
magnitude  and  difference  scores 
outperformed  both  the  ratio  and  actual 
review  scores. 

E.  Arkansas  Conclusion 

The  graphs  show  how  all  three  score 
functions  present  an  improvement  over  the 
actual  review  order.  The  difference  and 
magnitude  score  functions  are  comparable 
when  determining  review  order  for  farm 


records  with  historical  data.  Difference  and 
magnitude  scores  both  outperform  ratio 
scores.  However,  scoring  and  reviewing 
distinct  groups  of  records  decrease  the 
efficiency  of  the  score  functions. 

VI  Recommendations 

As  a  result  of  this  analysis,  the 
recommendation  is  to  use  the  magnitude 
score  function  for  the  2002  Census  of 
Agriculture.  However,  it  is  important  to 
note  that  batch  processing  will  decrease  the 
efficiency  of  this  function. 

The  magnitude  score  function  provides  the 
following  advantages: 

a)  It  does  not  require  using  record-level  data 
from  the  previous  census.  It  only  requires 
the  use  of  county  totals,  which  are  readily 
available.  This  is  very  important  since  for 
both  Michigan  and  Arkansas,  a  substantial 
number  of  records  did  not  have  historical 
data. 

b)  The  fact  that  the  magnitude  score  function 
only  uses  county  level  totals  adds  to  its 
simplicity  and  ease  of  implementation.  It 
will  be  easier  to  program  and  will  take  less 
time  to  run  during  production,  since  record- 
by-record  file  matching  will  not  be  needed. 

c)  The  magnitude  score  function  eliminates 
concerns  over  space  allocation  since  there 
will  be  no  need  to  maintain  historical  record- 
level  data  to  support  it. 

d)  Ultimately,  there  will  be  an  improvement 
in  the  overall  quality  of  the  final  estimates 
since  resources  will  be  used  for  a  more 
extensive  follow-up  of  the  most  important 
records  as  discussed  in  the  Lawrence  and 
McKenzie  (2000)  article. 
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VII  Further  Research 

The  current  plan  is  to  use  data  from  the  1997 
final  file  to  address  issues  that  need  further 
examination,  including  but  not  limited  to  the 
following: 

i.  Should  we  use  a  unique  list  of 
variables  for  each  state  or  a  generic 
list  of  variables  for  all  states?  For 
example,  broilers  were  used  to  score 
the  records  in  Arkansas  but  not 
Michigan,  potatoes  were  used  to 
score  the  records  in  Michigan  but  not 
Arkansas.  To  simplify 
implementation,  should  we  use  a 
generic  list  which  includes  broilers, 
and  potatoes  regardless  of  the  state 
under  review? 

ii.  Whether  we  use  a  generic  or  unique 
list  of  variables,  which  variables 
should  be  used  to  score  records?  In 
other  words,  should  we  focus  on  only 
the  major  commodities  in  the  state  or 
include  specialty  items  as  well? 

iii.  Is  the  score  function  identifying 
records  which  are  important  to  that 
particular  state? 

iv.  Should  we  use  a  permanent  file  with 
county  totals  for  every  item  of 
interest  or  should  county  totals  be 
created  on  the  fly  during  production? 
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Graph  1:  Scores  Calculated  for  Michigan  Records  With  Historical  Data  Graph  2:  Scores  Calculated  for  Michigan  Records  Without  Historical  Data 
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Graph  3:  Scores  Calculated  for  Michigan  Records  With  and  Without  Historical  Data  Graph  4:  Scores  Calculated  for  Michigan  Records  With  and  Without  Historical  Data 
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Graph  5:  Scores  Calculated  for  Arkansas  Records  With  Historical  Data  Graph  6:  Scores  Calculated  for  Arkansas  Records  Without  Historical  Data 
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Graph  7:  Scores  Calculated  for  Arkansas  Records  With  and  Without  Historical  Data  Graph  8:  Scores  Calculated  for  Arkansas  Records  With  and  Without  Historical  Data 

by  Edit  Batch 
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